Facial movement based avatar animation

ABSTRACT

Avatars are animated using predetermined avatar images that are selected based on facial features of a user extracted from video of the user. A user&#39;s facial features are tracked in a live video, facial feature parameters are determined from the tracked features, and avatar images are selected based on the facial feature parameters. The selected images are then displayed are sent to another device for display. Selecting and displaying different avatar images as a user&#39;s facial movements change animates the avatar. An avatar image can be selected from a series of avatar images representing a particular facial movement, such as blinking. An avatar image can also be generated from multiple avatar feature images selected from multiple avatar feature image series associated with different regions of a user&#39;s face (eyes, mouth, nose, eyebrows), which allows different regions of the avatar to be animated independently.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation application of U.S.application Ser. No. 13/997,271, entitled “FACIAL MOVEMENT BASED AVATARANIMATION,” now U.S. Pat. No. 9,466,142, which was filed on Jun. 23,2013, and which is a national stage entry under 35 U.S.C. §371(b) ofInternational Application No. PCT/CN12/86739, which was filed on Dec.12, 2012.

BACKGROUND

Users can be represented in software applications and various platforms,such as gaming or social media platforms, by an avatar. Some of theseavatars can be animated.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an exemplary environment in which technologiesdescribed herein can be implemented.

FIG. 2 illustrates two exemplary avatar images series for animating anavatar's face.

FIG. 3 illustrates two exemplary avatar feature image series and anavatar background.

FIG. 4 shows a graph illustrating exemplary linear and nonlinearrelationships between a facial feature parameter and an index to anavatar image series or an avatar feature image series.

FIG. 5 is a block diagram of a first exemplary computing device foranimating an avatar.

FIG. 6 is a flowchart of a first exemplary avatar animation method.

FIG. 7 is a flowchart of a second exemplary avatar animation method.

FIG. 8 is a block diagram of a second exemplary computing device foranimating an avatar.

FIG. 9 is a block diagram of an exemplary processor core that canexecute instructions as part of implementing technologies describedherein.

DETAILED DESCRIPTION

The technologies described herein allow avatars to be animated in realtime based on a user's facial or head movements (hereinafter,collectively referred to as “facial movements”). A user's facialfeatures are tracked in live video of the user, and facial featureparameters determined from the tracked features are mapped topredetermined avatar images. The disclosed avatar animation technologiesconsume less power than approaches that comprise generating or deforminga complex 3D avatar model based on a user's facial movements andrendering an avatar image based on such a model. Accordingly, thebattery life of mobile computing devices employing the disclosedtechnologies can be extended relative to devices that use morecomputationally intensive approaches.

Reference is now made to the drawings, wherein like reference numeralsare used to refer to like elements throughout. In the followingdescription, for purposes of explanation, numerous specific details areset forth in order to provide a thorough understanding thereof. It maybe evident, however, that the novel embodiments can be practiced withoutthese specific details. In other instances, known structures and devicesare shown in block diagram form in order to facilitate a descriptionthereof. The intention is to cover all modifications, equivalents, andalternatives within the scope of the claims.

FIG. 1 is a diagram of an exemplary environment 100 in whichtechnologies described herein can be implemented. The environment 100comprises a first user 110 using a first computing device 120 tocommunicate with a second user 130 using a second computing device 140via a network (or cloud) 150 via a video call or videoconference. Thefirst and second computing devices 120 and 140 can be any computingdevice, such as a mobile device (e.g., smartphone, laptop or tabletcomputer), desktop computer or server; and the network 150 can be anytype of network such as a Local Area Network (LAN), Wide Area Network(WAN) or the Internet.

A user interface 160 of a video call application running on the firstcomputing device 120 can comprise an avatar image 170 of the second user130 that occupies a majority of the user interface 160. Optionally, theuser interface 160 can also comprise an avatar image 130 of the user 110that occupies a smaller portion of the interface. The avatar images 170and 180 are based on facial features of the users 110 and 130 extractedfrom live video of the users to allow for real time communicationbetween the parties. That is, video frames are made available for videoavatar animation processing as soon as they are generated by a videocapture device. In some embodiments, where real time communicationbetween users is not required, an avatar can be animated based on storedvideo captured at any previous time.

Video can be provided to a computing device by, for example, a cameraintegrated into the computing device, such as a front-facing cameraintegrated into a smartphone (e.g., cameras 190 and 195) or tabletcomputer, or an external video capture device communicatively coupledwith the computing device, such as a wireless camcorder coupled to alaptop or a web camera coupled to a desktop computer.

In general, avatars are animated based on facial feature parametersdetermined from facial features tracked in video of a user. Exemplaryfacial features include the position of the upper and lower lips, upperand lower eyelids, mouth corners, left and right eyebrows (inside end,middle and outside end), chin, left and right ears, nose tip and leftand right nose wing. Exemplary facial feature parameters include thedegree of head rotation, degree of head tilt, distance between upper andlower lips, distance between mouth corners, distance between upper lipand nose tip, distance between nose wing and nose tip, distance betweenupper and lower eyelids, distance between eyebrow tips and distancebetween eyebrow tip and eyebrow middle. Facial features and facialfeature parameters can be tracked and determined in addition to thoselisted above.

Determined facial feature parameters are used to select predeterminedavatar images for animating an avatar. The avatar images arepredetermined in that they have been generated before a user's facialfeature parameters are determined from a video. The predetermined avatarimages can come pre-installed on a purchased computing device orinstalled at the device after purchase. Avatar images can be installedafter purchase by, for example, downloading a communication applicationthat supports avatar animation using techniques described herein or bydownloading avatar images separately. In addition, avatar images can beprovided by another computing device. For example, avatar images can beprovided to a computing device by a remote computing device with whichthe computing device is in communication. For instance, avatar imagescan be provided by a remote computing device as part of setting up avideo call or during the video call.

Predetermined avatar images can take various forms. For example, theycan be simple cartoon images or images generated from sophisticated 3Dmodels created using professional rendering engines. The predeterminedavatar images are also typically stored locally at a computing device.This allows for the quick provision of avatar images to other resourcesat the computing device and to other computing devices, such as remotecomputing devices participating in a video call with the computingdevice.

In some embodiments, an avatar is animated by selecting an avatar imagefrom one or more series of avatar images based at least in part onfacial features parameters determined from video of a user. As a user'sfacial features change in a video due to the user's changing facialmovements, different avatar images are selected and displayed, resultingin an animated avatar whose appearance corresponds to the facialmovements of the user. The manners in which an avatar can be animatedcan be based on the series of avatar images available to a computingdevice.

FIG. 2 illustrates two exemplary avatar image series 200 and 210 foranimating an avatar. Series 200 can be used to animate the opening andclosing of an avatar's mouth and series 210 can be used to animate anavatar blinking. Series 200 and 210 each comprise ten avatar imageshaving corresponding indices 220 and 230, respectively. An avatar imageseries (or avatar feature image series, as discussed below) can compriseany number of images. The number of images in a series can be based on,for example, a desired level of animation smoothness and a desiredamount of memory that avatar image series can occupy. In general, thefacial features tracked in a particular implementation of the disclosedtechnologies can be based on the avatar images series available to thedevice. For example, if a computing device only has access to the series200 and 210, the device may only track facial features relating to themouth and eyes.

Selecting an avatar image in a series can comprise mapping one or morefacial feature parameters to an image index. For example, the distancebetween a user's upper and lower lips can be mapped to one of theindices 220 of series 200. Various mapping approaches can be used. Oneexemplary mapping comprises normalizing a facial feature parameter to arange of zero to one and then performing a linear mapping of thenormalized parameter to a series index. For example, the distancebetween a user's upper and lower lips can be normalized to a range ofzero to one and then rounded up to the nearest tenth to generate anindex for the series 200. In some embodiments, an index can be generatedbased on two or more facial features parameters. For example, acombination of the distance between a user's upper and lower lips andthe distance between the corners of the user's mouth can be mapped to anindex of the series 200.

Accordingly, an avatar can be animated to open and close its mouth bysuccessively displaying adjacent avatar images in the series 200 inincreasing order by index value, and then successively in decreasingorder by index value. As used herein, the term “adjacent” as it refersto an avatar image in a series means an image that is the next orpreceding image in the animation sequence represented by the series. Forexample, with reference to avatar image series 200, avatar image 240 isadjacent to avatar images 250 and 260. However, it is not necessary thatadjacent avatar images be displayed in succession. For example, adisplayed avatar animation could comprise avatar image 260 (index=7)being displayed immediately after avatar image 250 (index=5) isdisplayed.

Additional avatar images series can be used to animate an avatar withfacial movements other than blinking and opening and closing its mouth.For example, avatar images series can be used to make an avatar yawn,smile, wink, raise its eyebrows, frown, etc.

In some embodiments, an avatar is animated using one avatar image seriesat a time. For example, if a computing device has access to avatar imageseries 200 and 210, the avatar can be made to blink or to open and closeits mouth, but cannot be made to do both simultaneously. In otherembodiments, an avatar image series can animate multiple facialmovements. For example, the series 200 and 210 can be combined to createan avatar image series comprising 100 avatar images corresponding tocombinations of the ten mouth images in the series 200 with the ten eyeimages in the series 210. Such an expanded avatar image series can beused to animate an avatar that can blink and open and close its mouth atthe same time. Avatar image series can animate more than two facialexpressions.

If a computing device has access to more than one avatar images seriesthat animate the same region of the face (e.g., two series that cananimate the mouth—one series that animates a smile and one series thatanimates a frown), the computing device can select which series to useto animate the avatar based on determined facial feature parameters. Forexample, a vertical distance between the mouth corners and the lower lipcan be used to determine if the user is smiling or frowning.

In some embodiments, multiple series of avatar feature images can beused to animate various regions of an avatar independently. Each avatarfeature image series corresponds to one region of an avatar's face(eyes, eyebrows, nose, mouth, etc.). In such embodiments, an avatarimage can be generated by combining the avatar feature images selectedfrom the avatar feature image series. An avatar feature image can beselected from a series based on facial feature parameters associatedwith the facial region corresponding with the series. In someembodiments, selected avatar feature images can be combined with anavatar background (e.g., an image of a face missing one or more parts)to generate an avatar image. Utilizing multiple series of avatar featureimages allows for independent animation of multiple regions of anavatar's face without the need for a single avatar image seriescontaining a large number of images.

FIG. 3 illustrates two exemplary avatar feature image series 300 and 310and an avatar background 320. The series 300 and 310 can be used toindependently animate the mouth and eyes of an avatar. The series 300can be used to animate an avatar's mouth opening and closing, and theseries 310 can be used to animate an avatar's eyes blinking. The avatarbackground 320 comprises an image of an avatar face without the eyes andmouth. An avatar image to be presented in an animation can be generatedby selecting an avatar feature image from series 310 based on, forexample, the distance between a user's upper and lower eyelids,selecting an avatar feature image from series 300 based on, for example,the distance between the upper and lower lips, and combining theselected avatar feature images with the avatar background 320. Forinstance, if, at a particular moment in time in a video, the determineddistance between a user's upper and lower lips is mapped to index six ofthe series 300 and the determined distance between the user's upper andlower eyelids is mapped to index ten of the series 310, selected avatarfeature images 330 and 340 can be combined with the avatar background320 to generate the avatar image 350.

In some embodiments, separate avatar feature image series can be used toindependently animate the left and right eyebrows and the left and righteyes. Moreover, a single image series can comprise images used toanimate different facial movements for a facial region. For example, asingle avatar feature image series for animating an avatar's mouth canbe used to making the avatar yawn, smile, grin, frown, or make the mouthmovements associated with a language's phonemes. Such a more complexavatar feature image series can correspond to a plurality of facialfeatures parameters.

FIG. 4 shows a graph 400 illustrating exemplary linear and nonlinearrelationships, via curves 410 and 420, respectively, between a facialfeature parameter (or a combination of multiple facial featureparameters) and an index to an avatar image series or an avatar featureimage series. In some embodiments, the mapping can be nonlinear toemphasize movement of avatar features. For example, consider the avatarfeature image series 300 used for animating the opening and closing ofan avatar's mouth. Although the images range from a closed mouth (image360) to a fully opened mouth (image 370), during typical conversation auser's mouth movements may not vary over this entire range. Thus, alinear mapping can result in the user's mouth movements being mapped toa relatively narrow range of indices, such as two through six, in series300. Using a non-linear mapping, such as the mapping represented bycurve 420, the mouth movements made by a user during a typicalconversation can be made to map to a wider range of indices (e.g. onethrough eight in series 300) to emphasize mouth movement in the avatar.

Avatar image information sent by one computing device to anothercomputing device for the purposes of animating an avatar can takevarious forms. For example, in some embodiments, avatar imageinformation can comprise an avatar image for each frame in an avataranimation sequence. The avatar images can be sent in a known image fileformat (e.g., .jpg, .tiff, .bmp) or other format. In some embodiments,if the receiving computing device has access to an avatar image seriesassociated with the avatar to be animated, the avatar image informationcan comprise an index into an avatar image series. The receiving devicecan then retrieve the appropriate avatar image from the series fordisplay at the receiving device. Avatar image information can alsocomprise identifiers of avatar image series.

In other embodiments, avatar image information can comprise one or moreindices to one or more avatar feature image series, and the receivingcomputing device can combine the avatar feature images associated withthe indices into an avatar image for display at a receiving computingdevice display. Avatar image information can also comprise an indicatorfor an avatar background to be combined with avatar feature images. Invarious embodiments, avatar image information can comprise facialfeature parameters determined from video and the receiving computingdevice can map the received parameters to an avatar image or one or moreavatar feature images.

In some embodiments, avatar image information can be sent to a secondcomputing device via an intermediate computing device, such as acloud-based server. For example, a cloud-based server that is part of avideoconferencing service can receive avatar image information from acomputing device being used by a first participant to thevideoconference, and distribute the avatar image information to otherparticipants to the videoconference.

In some embodiments, a user's avatar is animated at a display of thecomputing device that tracked the user's facial features video and/ordetermined facial feature parameters from the tracked facial features.For example, with reference to FIG. 1, the first computing device 120generates video of the user 110 with camera 190, tracks the user'sfacial features, determines facial feature parameters based on thetracked features and presents an animated avatar 180 of the user 110 inthe user interface 160 of the first device. In some embodiments, avatarimage information is not sent to another computing device. For example,a gaming console may animate a user's game avatar based on a user'sfacial movements captured by a depth camera connected to the gamingconsole. The gaming console can present the animated game avatar in adisplay connected to the gaming console, such as a television.

FIG. 5 is a block diagram of a first exemplary computing device 500 foranimating an avatar. The computing device 500 comprises a display 510, acommunication module 520 to send avatar image information to anothercomputing device, a facial feature tracking module 530 to track facialfeatures in video of a user, a facial feature parameter module 540 todetermine facial feature parameters from tracked facial features, anavatar image generation module 550, and an avatar image series store 560to store avatar image series and/or avatar feature images series.

The avatar image generation module 550 can select avatar images oravatar feature images from image series based on determined facialfeature parameters. The avatar image generation module 550 can alsoselect multiple avatar feature images from multiple avatar feature imageseries and combine the selected avatar feature images with an avatarbackground to generate an avatar image. The avatar image generationmodule 550 can further provide the selected or generated avatar imagefor display, at, for example, the computing device 500 or anothercomputing device. Optionally, the computing device can comprise a videocamera 570 to capture video of a user. In some embodiments, thecomputing device 500 receives video from an external video source 580,such as a web camera or a cloud-based video source.

It is to be understood that FIG. 5 illustrates one example of a set ofmodules that can be included in a computing device. In otherembodiments, a computing device can have more or fewer modules thanthose shown in FIG. 5. Further, modules shown as separate in FIG. 5 canbe combined into a single module, or a single module shown in FIG. 5 canbe split into multiple modules. Moreover, any of the modules shown inFIG. 5 can be part of the operating system of the computing device 500,one or more software applications independent of the operating system,or operate at another software layer. The modules shown in FIG. 5 can beimplemented in software, hardware, firmware or combinations thereof. Acomputer device referred to as being programmed to perform a method canbe programmed to perform the method via software, hardware, firmware orcombinations thereof.

FIG. 6 is a flowchart of a first exemplary avatar animation method 600.The method 600 can be performed by for example, a smartphone running avideo call application in which the user is represented by an avatar onthe display of a remote computing device being used by another party tothe call. The avatar tracks the user's facial movements and thesmartphone stores multiple avatar feature image series to animate theavatar's eyebrows, left and right eyes and mouth independently. At aprocess act 610, one or more predetermined avatar feature images areselected from one or more pluralities of predetermined avatar featureimages using a first computing device based at least in part on one ormore facial feature parameters determined from video of a user. In theexample, the smartphone selects an avatar feature image for theeyebrows, the left eye, the right eye and the mouth, based on facialfeature parameters determined from video of the user captured by thesmartphone's integrated camera. At a process act 620, an avatar image isgenerated based at least in part on the one more selected predeterminedavatar feature images. In the example, the smartphone combines theselected eyebrow, left eye, right eye and mouth avatar feature imageswith an avatar background image associated with the user to generate anavatar image. At a process act 630, avatar image information is providedfor display. In the example, the avatar image is provided to smartphonedisplay resources for display of the avatar in a portion of thesmartphone's display so that the user can see how his avatar is animatedfor other parties to the call. The smartphone also provides the avatarimage to the computing device of the other party to the call.

FIG. 7 is a flowchart of a second exemplary avatar animation method 700.The method 700 can be performed by, for example, a tablet computerexecuting a video call application in which the user is represented byan avatar on the display of a remote computing device operated by theother party to the video call. The avatar tracks the user's facialmovements and the smartphone stores multiple avatar image series used toanimate the avatar. The various avatar image series animate variousfacial movements of the user, such as smiling, frowning and blinking. Ata process act 710, a predetermined avatar image is selected from aplurality of predetermined avatar images at a first computing devicebased at least in part on one or more facial feature parametersdetermined from video of a user. In the example, an avatar image isselected from an avatar image series used to make the avatar smile,based on facial features parameters determined by video of the usercaptured by the tablet computer's integrated camera. At a process act720, the selected avatar image is displayed at a display of the firstcomputing device or avatar image information is sent to a secondcomputing device. In the example, the tablet sends the selected avatarimage to the second computing device.

The technologies described herein have at least the following exemplaryadvantages. The use of predetermined avatar images to animate an avatarprovides a lower power avatar animation option relative to animationapproaches that generate or deform a 3D avatar model on the fly based ontracked facial features or that use a sophisticated 3D rendering engineto generate the avatar image to be presented at a display. By avoidingsuch computationally expensive approaches, the technologies describedherein can also generate avatar animation more quickly. Although avataranimation has been discussed primarily in the context of video callapplications, the described technologies can be used in any scenarioswhere avatars are or can be animated, such as in gaming applications(e.g., console-based application or massively multiplayer onlinerole-playing games).

The technologies described herein can be performed by any of a varietyof computing devices, including mobile devices (such as smartphones,handheld computers, tablet computers, laptop computers, media players,portable gaming consoles, cameras and video recorders), non-mobiledevices (such as desktop computers, servers, stationary gaming consoles,smart televisions) and embedded devices (such as devices incorporatedinto a vehicle). As used herein, the term “computing devices” includescomputing systems and includes devices comprising multiple discretephysical components.

FIG. 8 is a block diagram of a second exemplary computing device 800 foranimating an avatar. Generally, components shown in FIG. 8 cancommunicate with other shown components, although not all connectionsare shown, for ease of illustration. The device 800 is a multiprocessorsystem comprising a first processor 802 and a second processor 804 andis illustrated as comprising point-to-point (P-P) interconnects. Forexample, a point-to-point (P-P) interface 806 of the processor 802 iscoupled to a point-to-point interface 807 of the processor 804 via apoint-to-point interconnection 805. It is to be understood that any orall of the point-to-point interconnects illustrated in FIG. 8 can bealternatively implemented as a multi-drop bus, and that any or all busesillustrated in FIG. 8 could be replaced by point-to-point interconnects.

As shown in FIG. 8, the processors 802 and 804 are multicore processors.Processor 802 comprises processor cores 808 and 809, and processor 804comprises processor cores 810 and 811. Processor cores 808-811 canexecute computer-executable instructions in a manner similar to thatdiscussed below in connection with FIG. 9, or in other manners.

Processors 802 and 804 further comprise at least one shared cache memory812 and 814, respectively. The shared caches 812 and 814 can store data(e.g., instructions) utilized by one or more components of theprocessor, such as the processor cores 808-809 and 810-811. The sharedcaches 812 and 814 can be part of a memory hierarchy for the device 800.For example, the shared cache 812 can locally store data that is alsostored in a memory 816 to allow for faster access to the data bycomponents of the processor 802. In some embodiments, the shared caches812 and 814 can comprise multiple cache layers, such as level 1 (L1),level 2 (L2), level 3 (L3), level 4 (L4), and/or other caches or cachelayers, such as a last level cache (LLC).

Although the device 800 is shown with two processors, the device 800 cancomprise only one processor or more than two processors. Further, aprocessor can comprise one or more processor cores. A processor can takevarious forms such as a central processing unit, a controller, agraphics processor, an accelerator (such as a graphics accelerator ordigital signal processor (DSP)) or a field programmable gate array(FPGA). A processor in a device can be the same as or different fromother processors in the device. In some embodiments, the device 800 cancomprise one or more processors that are heterogeneous or asymmetric toa first processor, accelerator, FPGA, or any other processor. There canbe a variety of differences between the processing elements in a systemin terms of a spectrum of metrics of merit including architectural,microarchitectural, thermal, power consumption characteristics and thelike. These differences can effectively manifest themselves as asymmetryand heterogeneity amongst the processors in a system. In someembodiments, the processors 802 and 804 reside in the same die package.

Processors 802 and 804 further comprise memory controller logic (MC) 820and 822. As shown in FIG. 8, MCs 820 and 822 control memories 816 and818 coupled to the processors 802 and 804, respectively. The memories816 and 818 can comprise various types of memories, such as volatilememory (e.g., dynamic random access memories (DRAM), static randomaccess memory (SRAM)) or non-volatile memory (e.g., flash memory). WhileMCs 820 and 822 are illustrated as being integrated into the processors802 and 804, in alternative embodiments, the MCs can be logic externalto a processor, and can comprise one or more layers of a memoryhierarchy.

Processors 802 and 804 are coupled to an Input/Output (I/O) subsystem830 via P-P interconnections 832 and 834. The point-to-pointinterconnection 832 connects a point-to-point interface 836 of theprocessor 802 with a point-to-point interface 838 of the I/O subsystem830, and the point-to-point interconnection 834 connects apoint-to-point interface 840 of the processor 804 with a point-to-pointinterface 842 of the I/O subsystems 30. Input/Output subsystem 830further includes an interface 850 to couple I/O subsystem 830 to agraphics engine 852, which can be a high-performance graphics engine.The I/O subsystem 830 and the graphics engine 852 are coupled via a bus854. Alternately the bus 844 could be a point-to-point interconnection.

Input/Output subsystem 830 is further coupled to a first bus 860 via aninterface 862. The first bus 860 can be a Peripheral ComponentInterconnect (PCI) bus, a PCI Express bus, another third generation I/Ointerconnection bus or any other type of bus.

Various I/O devices 864 can be coupled to the first bus 860. A busbridge 870 can couple the first bus 860 to a second bus 880. In someembodiments, the second bus 880 can be a low pin count (LPC) bus.Various devices can be coupled to the second bus 880 including, forexample, a keyboard/mouse 882, audio I/O devices 888 and a storagedevice 890, such as a hard disk drive, solid-state drive or otherstorage device for storing computer-executable instructions (code) 292.The code 292 comprises computer-executable instructions for performingtechnologies described herein. Additional components that can be coupledto the second bus 880 include communication device(s) 884, which canprovide for communication between the device 800 and one or more wiredor wireless networks 886 (e.g. Wi-Fi, cellular or satellite networks)via one or more wired or wireless communication links (e.g., wire,cable, Ethernet connection, radio-frequency (RF) channel, infraredchannel, Wi-Fi channel) using one or more communication standards (e.g.,IEEE 802.11 standard and its supplements).

The device 800 can comprise removable memory such flash memory cards(e.g., 3D (Secure Digital) cards), memory sticks, Subscriber IdentityModule (SIM) cards). The memory in device 800 (including caches 812 and814, memories 816 and 818 and storage device 890) can store data and/orcomputer-executable instructions for executing an operating system 894and application programs 296. Example data includes web pages, textmessages, images, sound files, video data, avatar image series, avatarfeature images series, avatar backgrounds or other data sets to be sentto and/or received from one or more network servers or other devices bythe device 800 via one or more wired or wireless networks, or for use bythe device 800. The device 800 can also have access to external memory(not shown) such as external hard drives or cloud-based storage.

The operating system 894 can control the allocation and usage of thecomponents illustrated in FIG. 8 and support one or more applicationprograms 896. The application programs 896 can include common mobilecomputing device applications (e.g., email applications, calendars,contact managers, web browsers, messaging applications) as well as othercomputing applications, such as a video call application 897 thatanimates user avatars based on a user's facial features.

The device 800 can support various input devices, such as a touchscreen, microphone, camera, physical keyboard, proximity sensor andtrackball and one or more output devices, such as a speaker and adisplay. Other possible input and output devices include piezoelectricand other haptic I/O devices. Any of the input or output devices can beinternal to, external to or removably attachable with the device 800,such as an external depth camera or a connected television. Externalinput and output devices can communicate with the device 800 via wiredor wireless connections.

In addition, the computing device 800 can provide one or more naturaluser interfaces (NUIs). For example, the operating system 892 orapplications 894 can comprise speech recognition logic as part of avoice user interface that allows a user to operate the device 800 viavoice commands Further, the device 800 can comprise input devices andlogic that allows a user to interact with the device 800 via a body handor face gestures. For example, a user's hand gestures can be detectedand interpreted to provide input to a gaming application.

The device 800 can further comprise one or more wireless modems (whichcould comprise communication devices 884) coupled to one or moreantennas to support communication between the system 800 and externaldevices. The wireless modems can support various wireless communicationprotocols and technologies such as Near Field Communication (NFC),Wi-Fi, Bluetooth, 4G Long Term Evolution (LTE), Code DivisionMultiplexing Access (CDMA), Universal Mobile Telecommunication System(UMTS) and Global System for Mobile Telecommunication (GSM). Inaddition, the wireless modems can support communication with one or morecellular networks for data and voice communications within a singlecellular network, between cellular networks, or between the mobilecomputing device and a public switched telephone network (PSTN).

The device 800 can further include at least one input/output port (whichcan be, for example, a USB port IEEE 994 (FireWire) port, and/or RS-232port) comprising physical connectors, a power supply, a satellitenavigation system receiver such as a GPS receiver, a gyroscope, anaccelerometer and a compass. A GPS receiver can be coupled to a GPSantenna. The device 800 can further include one or more additionalantennas coupled to one or more additional receivers, transmittersand/or transceivers to enable additional functions.

It is to be understood that FIG. 8 illustrates only one exemplarycomputing device architecture. Computing devices based on alternativearchitectures can be used to implement technologies described herein.For example, instead of the processors 802 and 804, and the graphicsengine 852 being located on discrete integrated circuits, a computingdevice can comprise a SoC (system-on-a-chip) integrated circuitincorporating multiple processors, a graphics engine and additionalcomponents. Further, a computing device can connect elements via busconfigurations different from that shown in FIG. 8. Moreover, theillustrated components in FIG. 8 are not required or all-inclusive, asshown components can be removed and other components added inalternative embodiments.

FIG. 9 is a block diagram of an exemplary processor core 900 to executecomputer-executable instructions for implementing technologies describedherein. The processor core 900 can be a core for any type of processor,such as a microprocessor, an embedded processor, a digital signalprocessor (DSP) or a network processor. The processor core 900 can be asingle-threaded core or a multithreaded core in that it may include morethan one hardware thread context (or “logical processor”) per core.

FIG. 9 also illustrates a memory 910 coupled to the processor 900. Thememory 910 can be any memory described herein or any other memory knownto those of skill in the art. The memory 910 can storecomputer-executable instruction 915 (code) executable by the processorcore 900.

The processor core comprises front-end logic 920 that receivesinstructions from the memory 910. An instruction can be processed by oneor more decoders 930. The decoder 930 can generate as its output a microoperation such as a fixed width micro operation in a predefined formator generate other instructions, microinstructions, or control signals,which reflect the original code instruction. The front-end logic 920further comprises register renaming logic 935 and scheduling logic 940,which generally allocate resources and queues operations correspondingto converting an instruction for execution.

The processor core 900 further comprises execution logic 950, whichcomprises one or more execution units (EUs) 965-1 through 965-N. Someprocessor core embodiments can include a number of execution unitsdedicated to specific functions or sets of functions. Other embodimentscan include only one execution unit or one execution unit that canperform a particular function. The execution logic 950 performs theoperations specified by code instructions. After completion of executionof the operations specified by the code instructions, back-end logic 970retires instructions using retirement logic 975. In some embodiments,the processor core 900 allows out of order execution but requiresin-order retirement of instructions. Retirement logic 970 can take avariety of forms as known to those of skill in the art (e.g., re-orderbuffers or the like).

The processor core 900 is transformed during execution of instructions,at least in terms of the output generated by the decoder 930, hardwareregisters and tables utilized by the register renaming logic 935, andany registers (not shown) modified by the execution logic 950. Althoughnot illustrated in FIG. 9, a processor can include other elements on anintegrated chip with the processor core 900. For example, a processormay include additional elements such as memory control logic, one ormore graphics engines, I/O control logic and/or one or more caches.

Referring back to FIG. 1, the network or cloud 150 can provide variouscloud-based services that can be used to implement technologiesdescribed herein. For example, avatar image series or avatar featureimages series or applications that employ avatar animation techniquesdescribed herein can be provided by cloud-based services.

Any of the disclosed methods can be implemented as computer-executableinstructions or a computer program product. Such instructions can causea computer to perform any of the disclosed methods. Generally, as usedherein, the term “computer” refers to any computing device or systemdescribed or mentioned herein, or any other computing device. Thus, theterm “computer-executable instruction” refers to instructions that canbe executed by any computing device described or mentioned herein, orany other computing device.

The computer-executable instructions or computer program products aswell as any data created and used during implementation of the disclosedtechnologies can be stored on one or more tangible computer-readablestorage media, such as optical media discs (e.g., DVDs, CDs), volatilememory components (e.g., DRAM, SRAM), or non-volatile memory components(e.g., flash memory, disk drives). Computer-readable storage media canbe contained in computer-readable storage devices such as solid-statedrives, USB flash drives, and memory modules. Alternatively, thecomputer-executable instructions may be performed by specific hardwarecomponents that contain hardwired logic for performing all or a portionof disclosed methods, or by any combination of computer-readable storagemedia and hardware components.

The computer-executable instructions can be part of, for example, adedicated software application or a software application that isaccessed via a web browser or other software application (such as aremote computing application). Such software can be executed, forexample, on a single computing device or in a network environment usingone or more network computers. Further, it is to be understood that thedisclosed technology is not limited to any specific computer language orprogram. For instance, the disclosed technologies can be implemented bysoftware written in C++, Java, Perl, JavaScript, Adobe Flash, or anyother suitable programming language. Likewise, the disclosedtechnologies are not limited to any particular computer or type ofhardware. Certain details of suitable computers and hardware are knownand need not be set forth in detail in this disclosure.

Furthermore, any of the software-based embodiments (comprising, forexample, computer-executable instructions for causing a computer toperform any of the disclosed methods) can be uploaded, downloaded orremotely accessed through a suitable communication means. Such suitablecommunication means include, for example, the Internet the World WideWeb, an intranet, cable (including fiber optic cable), magneticcommunications, electromagnetic communications (including RF, microwave,and infrared communications), electronic communications, or other suchcommunication means.

As used in this application and in the claims, a list of items joined bythe term “and/or” can mean any combination of the listed items. Forexample, the phrase “A, B and/or C” can mean A; B; C; A and B; A and C;B and C; or A, B and C. As used in this application and in the claims, alist of items joined by the terms “at least one of” or “one or more of”can mean any combination of the listed terms. For example, the phrases“at least one of A, B or C” or “one or more of A, B or C” can mean A; B;C; A and B; A and C; B and C; or A, B and C.

The disclosed methods, apparatuses and systems are not to be construedas limiting in anyway. Instead, the present disclosure is directedtoward all novel and nonobvious features and aspects of the variousdisclosed embodiments, alone and in various combinations andsubcombinations with one another. The disclosed methods, apparatuses,and systems are not limited to any specific aspect or feature orcombination thereof, nor do the disclosed embodiments require that anyone or more specific advantages be present or problems be solved.

Theories of operation, scientific principles or other theoreticaldescriptions presented herein in reference to the apparatuses or methodsof this disclosure have been provided for the purposes of betterunderstanding and are not intended to be limiting in scope. Theapparatuses and methods in the appended claims are not limited to thoseapparatuses and methods that function in the manner described by suchtheories of operation.

Although the operations of some of the disclosed methods are describedin a particular, sequential order for convenient presentation, it is tobe understood that this manner of description encompasses rearrangement,unless a particular ordering is required by specific language set forthherein. For example, operations described sequentially may in some casesbe rearranged or performed concurrently. Moreover, for the sake ofsimplicity the attached figures may not show the various ways in whichthe disclosed methods can be used in conjunction with other methods.

The following examples pertain to further embodiments.

EXAMPLE 1

An avatar animation method, comprising: selecting one or morepredetermined avatar feature images from one or more pluralities ofpredetermined avatar feature images using a first computing device basedat least in part on one or more facial feature parameters determinedfrom video of a user; generating an avatar image based at least in parton the one more selected predetermined avatar feature images; andproviding avatar image information for display.

EXAMPLE 2

The method of Example 1, further comprising displaying the avatar imageat a display of the first computing device.

EXAMPLE 3

The method of Example 1, wherein the providing comprises sending avatarimage information to a second computing device.

EXAMPLE 4

The method of Example 1, wherein the selecting comprises, for respectiveones of the one or more facial feature parameters, mapping therespective facial feature parameter to the selected predetermined avatarfeature image belonging to the plurality of predetermined avatarfeatured images associated with the respective facial feature parameter.

EXAMPLE 5

The method of Example 4, wherein the mapping is a nonlinear mapping.

EXAMPLE 6

The method of Example 1, wherein the avatar image is further based atleast in part on an avatar background.

EXAMPLE 7

The method of Example 1, further comprising displaying the avatar imageat a display of the second computing device.

EXAMPLE 8

An avatar animation method, comprising: selecting one or morepredetermined avatar feature images from one or more pluralities ofpredetermined avatar feature images using the computing device based atleast in part on one or more facial feature parameters determined fromvideo of a user; generating an avatar image based at least in part onthe one more selected predetermined avatar feature images; and providingavatar image information for display.

EXAMPLE 9

The method of Example 8, further comprising displaying the avatar imageat a display of the first computing device.

EXAMPLE 10

The method of Example 8, wherein the providing comprises sending avatarimage information to a second computing device.

EXAMPLE 11

The method of Example 10, further comprising displaying the avatar imageat a display of the second computing device.

EXAMPLE 12

The method of Example 8, wherein the selecting comprises, for respectiveones of the one or more facial feature parameters, mapping therespective facial feature parameter to the selected predetermined avatarfeature image belonging to the plurality of predetermined avatarfeatured images associated with the respective facial feature parameter.

EXAMPLE 13

The method of Example 8, wherein the mapping is a nonlinear mapping.

EXAMPLE 14

The method of Example 8, wherein the avatar image is further based atleast in part on an avatar background.

EXAMPLE 15

The method of Example 8, further comprising displaying the avatar imageat a display of the second computing device.

EXAMPLE 16

A method of distributing computer-executable instructions for causing acomputing device to perform an avatar animation method, the methodcomprising: sending computer-readable instructions to a computing devicefor causing the computing device to a perform an avatar animationmethod, the method comprising: selecting one or more predeterminedavatar feature images from one or more pluralities of predeterminedavatar feature images using a first computing device based at least inpart on one or more facial feature parameters determined from video of auser; generating an avatar image based at least in part on the one moreselected predetermined avatar feature images; and providing avatar imageinformation for display, and storing the computer-readable instructionsat the computing device.

EXAMPLE 17

One or more computer-readable storage media storing computer-executableinstructions for causing a computing device to perform any one of theavatar animation methods of Examples 1-16.

EXAMPLE 18

At least one computing device programmed to perform any one of theavatar animation methods of Examples 1-16.

EXAMPLE 19

At least one computing device comprising a means to perform any one ofthe methods of Examples 1-16.

EXAMPLE 20

An avatar animation method, comprising: tracking one or more facialfeatures in video of a user; and sending avatar image information to asecond computing device.

EXAMPLE 21

The method of Example 20, wherein the avatar image information comprisesone or more indices to one or more avatar feature image series or one ormore indices to one or more avatar image series.

EXAMPLE 22

The method of Example 20, where the avatar image information furthercomprises an avatar background image or an indicator of an avatarbackground image.

EXAMPLE 23

One or more computer-readable storage media storing computer-executableinstructions for causing a computing device to perform any one of theavatar animation methods of Examples 20-22.

EXAMPLE 24

At least one computing device programmed to perform any one of theavatar animation methods of Examples 20-22.

EXAMPLE 25

A computing device, comprising: a facial feature tracking module totrack one or more facial features in video of a user; a facial featureparameter module to determine one or more facial feature parameters fromthe one or more tracked facial features; and an avatar image generationmodule to select an avatar image from one or more avatar image seriesbased on the determined one or more facial feature parameters andprovide the selected avatar image for display.

EXAMPLE 26

A computing device, comprising: a facial feature tracking module totrack one or more facial features in video of a user; a facial featureparameter module to determine one or more facial feature parameters fromthe one or more tracked facial features; and an avatar image generationmodule to select one or more avatar feature images from one or moreavatar feature images series based on the determined one or more facialfeature parameters, generate an avatar image based on the one or moreselected avatar feature images, and provide the generated avatar imagefor display.

1. An avatar animation method, comprising: selecting one or morepredetermined avatar feature images from one or more pluralities ofpredetermined avatar feature images using a first computing device basedat least in part on one or more facial feature parameters determinedfrom video of a user; generating an avatar image based at least in parton the one more selected predetermined avatar feature images; andproviding avatar image information for display.
 2. The method of claim1, further comprising displaying the avatar image at a display of thefirst computing device.
 3. The method of claim 1, wherein the providingcomprises sending avatar image information to a second computing device.4. The method of claim 3, further comprising displaying the avatar imageat a display of the second computing device.
 5. The method of claim 1,wherein the selecting comprises, for respective ones of the one or morefacial feature parameters, mapping the respective facial featureparameter to the selected predetermined avatar feature image belongingto the plurality of predetermined avatar featured images associated withthe respective facial feature parameter.
 6. The method of claim 5,wherein the mapping is a nonlinear mapping.
 7. The method of claim 1,wherein the avatar image is further based at least in part on an avatarbackground.
 8. One or more computer-readable storage media storingcomputer-executable instructions for causing a computing device toperform an avatar animation method, the method comprising: selecting oneor more predetermined avatar feature images from one or more pluralitiesof predetermined avatar feature images using the computing device basedat least in part on one or more facial feature parameters determinedfrom video of a user; generating an avatar image based at least in parton the one more selected predetermined avatar feature images; andproviding avatar image information for display.
 9. The one or morecomputer-readable storage media of claim 8, further comprisingdisplaying the avatar image at a display of the first computing device.10. The one or more computer-readable storage media of claim 8, whereinthe providing comprises sending avatar image information to a secondcomputing device.
 11. The one or more computer-readable storage media ofclaim 8, wherein the selecting comprises, for respective ones of the oneor more facial feature parameters, mapping the respective facial featureparameter to the selected predetermined avatar feature image belongingto the plurality of predetermined avatar featured images associated withthe respective facial feature parameter.
 12. The one or morecomputer-readable storage media of claim 8, wherein the mapping is anonlinear mapping.
 13. The one or more computer-readable storage mediaof claim 8, wherein the avatar image is further based at least in parton an avatar background.
 14. One or more computer-readable storage mediastoring computer-executable instructions for causing a computing deviceto perform an avatar animation method, the method comprising: selectinga predetermined avatar image from a plurality of predetermined avatarfeature images at the computing device based at least in part on one ormore facial feature parameters determined from video of a user; anddisplaying the selected avatar image at a display of the computingdevice or sending the selected avatar image to a second computingdevice.
 15. The one or more computer-readable storage media of claim 14,further comprising repeating the selecting, and the displaying or thesending, such that successive avatar images displayed at the display orsent to the second computing device are adjacent predetermined avatarimages in the corresponding plurality of predetermined avatar images.16. The one or more computer-readable storage media of claim 14, whereinthe selecting comprises, mapping the one or more facial featureparameters to the selected predetermined avatar image.